Skip to content

Restructure all the configs#951

Draft
edmundmiller wants to merge 9 commits intomasterfrom
whole-refactor
Draft

Restructure all the configs#951
edmundmiller wants to merge 9 commits intomasterfrom
whole-refactor

Conversation

@edmundmiller
Copy link
Contributor

@edmundmiller edmundmiller commented Aug 29, 2025

See a smaller example in #854

This would need a refactor on the website per @mashehu so this needs to wait for that(if no one has objections to the structure)

edmundmiller and others added 9 commits March 4, 2025 20:21
- Rename conf/uppmax/uppmax.config to conf/uppmax/nextflow.config
- Rename conf/uppmax/pipeline/ to conf/uppmax/pipelines/
- Move pipeline-specific configs to their own subdirectories:
  - conf/pipeline/sarek/uppmax.config → conf/uppmax/pipelines/sarek/nextflow.config
  - conf/pipeline/ampliseq/uppmax.config → conf/uppmax/pipelines/ampliseq/nextflow.config
- Move pipeline documentation to co-located README files:
  - docs/pipeline/sarek/uppmax.md → conf/uppmax/pipelines/sarek/README.md
  - docs/pipeline/ampliseq/uppmax.md → conf/uppmax/pipelines/ampliseq/README.md
- Remove duplicate config and documentation files

All uppmax-related files are now co-located under conf/uppmax/ with
consistent naming conventions (nextflow.config, README.md).

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Update the includeConfig path for uppmax profile to point to the new
location at conf/uppmax/nextflow.config after directory reorganization.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Add bin/reorganize-cluster.sh script to automate the reorganization
of cluster configuration directories to follow the new structure:

- Renames cluster.config → nextflow.config
- Renames pipeline/ → pipelines/
- Moves pipeline configs to organized subdirectories
- Moves pipeline docs to co-located README files
- Updates nfcore_custom.config with correct paths
- Provides comprehensive logging and validation

Usage: ./bin/reorganize-cluster.sh <cluster_name>

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Reorganize sanger cluster configuration to new directory structure:
  - conf/sanger.config → conf/sanger/nextflow.config
  - Update nfcore_custom.config path reference
- Improve reorganize-cluster.sh script to handle single config files:
  - Create cluster directory if it doesn't exist
  - Move standalone .config files into directory structure
  - Support both existing directories and single files

This demonstrates the automation working for the common case where
clusters have a single config file rather than a complex directory.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Add comprehensive documentation to CLAUDE.md covering:

- New directory structure pattern with examples
- Automated migration process using reorganize-cluster.sh script
- Manual migration steps for edge cases
- Benefits of the new organization approach
- Critical registry update requirements

Includes examples for both complex (uppmax) and simple (sanger)
cluster configurations to guide future reorganizations.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Fix update-pipeline-configs.yml to create files in conf/pipeline/
  instead of pipeline/ to match current directory structure
- Fix typo "succesfully" → "successfully" in fix-linting.yml
- Fix duplicate step names in fix-linting.yml for better clarity
- Remove extra whitespace in conditional check

These fixes ensure automated pipeline config creation works correctly
with the reorganized directory structure.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Complete the mass migration of cluster configurations:

- Add mass migration automation script (migrate-all-clusters.sh)
- Migrate 134 clusters from conf/*.config to conf/*/nextflow.config
- Move pipeline-specific configs to organized subdirectories:
  - Move conf/pipeline/*/cluster.config → conf/cluster/pipelines/*/nextflow.config
  - Move docs/pipeline/*/cluster.md → conf/cluster/pipelines/*/README.md
- Update all registry paths in nfcore_custom.config automatically
- Remove all duplicate config and documentation files

This completes the infrastructure reorganization with:
- 137 total cluster directories (up from 3)
- All clusters now follow consistent structure pattern
- Co-located configuration and documentation
- Standardized naming (nextflow.config, README.md)
- 100% automated migration with zero manual fixes required

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@jfy133
Copy link
Member

jfy133 commented Aug 30, 2025

@edmundmiller I would need time to properly test this (which won't come for a while....unless we look during the retreat) but i. The meantime are the AI scripts meant to stay in here?

Also I can't see from my phone as too many files - but have you updated the auto generation action for new configs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

2 participants